<!DOCTYPE html PUBLIC "-//w3c//dtd html 4.0 transitional//en">
<html>
<head>
  <meta http-equiv="Content-Type"
 content="text/html; charset=iso-8859-1">
  <meta name="GENERATOR"
 content="Mozilla/4.7 [en]C-CCK-MCD NSCPCD47  (Win95; I) [Netscape]">
  <title>CorefEval</title>
</head>
<body text="#000000" bgcolor="#fff0f0" link="#ff0000" vlink="#800080"
 alink="#0000ff">
<h1>
<font face="Arial Alternative"><font color="#3333ff">CorefEval</font></font></h1>
<font color="#000000">The CorefEval utility computes a coreference
score using a version of Marc Vilain's MUC-6 coreference scoring
algorithm.&nbsp; It is invoked by<br>
<br>
</font>
<div style="margin-left: 40px;"><big><font style="font-weight: bold;"
 color="#000000"><span style="font-family: monospace;">jet -CorefEval</span></font></big><font
 color="#000000">&nbsp; <span style="font-style: italic;">system-response-collection
key-collection</span></font><br>
</div>
<font color="#000000"><br>
where <span style="font-style: italic;">system-response-collection </span>is
the system response and <span style="font-style: italic;">key-collection
</span>is the coreference key;&nbsp; both are collections of
XML-annotated files.&nbsp; Each collection file should contain a list
of the file names of the XML-annotated document
files, one file name per line (the two files should contain the same
number of lines).&nbsp; File names may either be absolute
paths or relative paths;&nbsp; relative paths are interpreted relative
to the directory containing the collection<span
 style="font-style: italic;">.</span><br>
<br>
</font><font color="#000000">The individual document files are marked
for coreference with the tags </font><font color="#000000"><span
 style="font-family: monospace;">&lt;mention entity=</span></font><font
 style="font-style: italic;" color="#000000"><span
 style="font-family: monospace;"></span>n</font><font color="#000000"><span
 style="font-family: monospace;">&gt; ... &lt;/mention&gt;</span></font><font
 color="#000000"><span style="font-family: monospace;"></span>, where <span
 style="font-style: italic;">n</span> is an integer.<br>
Sequences enclosed in tags with the same value of <span
 style="font-style: italic;">n</span> are coreferential.<br>
<br>
When the system response and key are compared, mentions match if the <span
 style="font-style: italic;">last character</span> of the mention in
the response and key are the same.&nbsp; Mentions in response or key
which do not have a corresponding mention in the other document are
ignored.&nbsp; The scorer reports a recall and precision score for each
document and for the entire collection.<br>
</font><font color="#000000"><br>
</font><br>
</body>
</html>
